What Works Clearinghouse™ 


U.S. DEPARTMENT OF EDUCATION 


111 WWC Intervention Report /joe 

a summary of findings from a systematic review of the evidence ® ^ 


Primary Mathematics 


systematic 


INSTITUTE of 

EDUCATION SCIENCES 


Updated December 2015 


Singapore Math® 

This intervention report presents findings from a systematic review of Singapore Math® conducted 
using the What Works Clearinghouse (WWC) Procedures and Standards Handbook, version 3.0, and 
the Primary Mathematics review protocol, version 3.1. No studies of Singapore Math® that fall within 
the scope of the Primary Mathematics review protocol meet WWC group design standards. Because 
no studies meet WWC group design standards at this time, the WWC is unable to draw any con- 
clusions based on research about the effectiveness or ineffectiveness of Singapore Math® on the 
achievement of primary students in kindergarten through grade 8. Research that meets WWC design 
standards is needed to determine the effectiveness or ineffectiveness of this intervention. 

Program Description 1 

Singapore Math® is a collection of mathematics curricula developed by Singapore’s Ministry of Education and pri- 
vate textbook publishers for use in Singapore schools. Singapore Math ® is comprised of Kindergarten Mathemat- 
ics, Primary Mathematics for students in grades 1-6, and Dimensions Mathematics for students in grades 7-8. The 
program is centered on problem solving, emphasizes computational skills, and focuses on conceptual understand- 
ing and strategic thinking. With these three components, Singapore Math® aims to provide more in-depth coverage 
of a relatively smaller number of topics than typical mathematics textbooks. Singapore Math® emphasizes problem- 
based development of mathematical concepts and uses concrete illustrations to show how to solve multistep 
problems. The content framework covers topics in increasingly advanced detail in successive grades. 

Research 2 

The WWC identified 17 studies of Singapore Math® for primary students that were published or released between 
1983 and 2014. 

Three studies are within the scope of the Primary Mathematics review protocol but do not meet WWC group design 
standards. 

• Two studies used a quasi-experimental design (QED) to assess the effects of Singapore Math®, but neither 
study established baseline equivalence between the intervention and comparison groups as required by WWC 
group design standards. 

• One study used a randomized controlled trial (RCT) to assess the effects of Singapore Math®. However, the 
analysis included students who entered study classrooms (non-randomly) after random assignment occurred, 
and the study did not establish baseline equivalence between the analytical intervention and comparison groups 
as required by WWC group design standards. 

Ten studies are out of the scope of the Primary Mathematics review protocol because they have an ineligible study 
design. These include studies without comparison groups or literature reviews or other publications that are not 
primary analyses of the effectiveness of Singapore Math®. 

Four studies are out of the scope of the Primary Mathematics review protocol for reasons other than study design. 
These include studies that did not examine a relevant outcome domain specified in the protocol— specifically, they 
did not examine outcomes on student mathematics achievement. Instead, studies examined other types of out- 
comes, such as ones related to curriculum implementation or teacher practices. 
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Endnotes 

1 The descriptive information for this program was obtained from a publicly available source: the program’s website (http://www. 
singaporemath.com, downloaded July 2014). The WWC requests developers to review the program description sections for accuracy 
from their perspective. The program description was provided to the developer in August 2014, and the WWC incorporated feedback 
from the developer. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. 

2 The literature search reflects documents publicly available by December 2014. The previous report was released under the Middle 
School Math topic area in April 2009. This report has been updated to include reviews of seven studies that were not reviewed in the 
previous report. Of the additional studies, four used an ineligible study design or were out of the scope of the protocol, and three were 
within the scope of the protocol but did not meet WWC group design standards. A complete list and disposition of all studies reviewed 
are provided in the references. The studies in this report were reviewed using the Standards from the WWC Procedures and Standards 
Handbook (version 3.0), along with those described in the Primary Mathematics review protocol (version 3.1). The evidence presented 
in this report is based on available research. Findings and conclusions may change as new research becomes available. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Intervention 
Intervention report 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Single-case design 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent of 
evidence levels are given in the WWC Procedures and Standards Handbook (version 3.0). 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Procedures and Standards Hand- 
book (version 3.0). 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 
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Glossary of Terms 


Standard deviation The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance Statistical significance is the probability that the difference between groups is a result of 

chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


Substantively important a substantively important finding is one that has an effect size of 0.25 or greater, regardless 

of statistical significance. 


Systematic review a review of existing literature on a topic that is identified and reviewed using explicit meth- 
ods. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their find- 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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